Matrix Outer Product

The matrix outer product takes two vectors and produces a matrix. If you have a column vector $u$ of size $m$ and a row vector $v$ of size $n$, their outer product $u \otimes v$ (or simply $u v^T$) is an $m \times n$ matrix.

This operation is particularly important in machine learning for tasks like updating weights in gradient descent, where we often compute the outer product of an error vector and an input vector to get the gradient matrix for the weights.

How it Works

Unlike the inner product (dot product) which results in a single scalar value, the outer product pairs every element of the first vector with every element of the second vector.

If we have vectors $u$ and $v$, the element at row $i$ and column $j$ in the resulting matrix is simply $u_i \times v_j$.

Here is an example visualization of a $2 \times 2$ matrix resulting from an outer product:

Outer Product Visualization

Interact with the vectors error and inputs to see how they form the outer product matrix.

Vector error (Column):

Vector inputs (Row):

weight_updates = error ⊗ inputs

Matrix View (weight_updates ∈ R^2ˣ^2)

error 1 × inputs

error 2 × inputs

Python Implementation

Using numpy, the outer product is efficiently computed using np.outer().

import numpy as np

# Let's say we have an error for 2 output nodes
error = np.array([5, 10])

# And we have an input vector of 2 features
x = np.array([2, 3])

# We compute the outer product to find the weight updates (gradients)
# Result will be a (2x2) matrix
weight_updates = np.outer(error, x)

print("Weight Updates Matrix:\n", weight_updates)

In neural networks, this tells us exactly how much each weight connecting the $j$-th input to the $i$-th output contributed to the error, allowing us to update the entire weight matrix efficiently!